Media decoding method based on cloud computing and decoder thereof

ABSTRACT

A media decoding method based on cloud computing and decoder thereof are provided by embodiments of the present invention, which are easy to use and applicable to a media of any form, and its requirement for computer resource is low. The method includes: extracting representing features from a media code stream to be decoded; searching in the cloud for a media object which has similar representing features with the media code stream to be decoded by using a feature matching method and the representing features extracted; filling, replacing and improving parts or segments of the media code stream to be decoded with whole or parts of the media object.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority from CN Patent Application Serial No.201310551685.4, filed on Nov. 7, 2013, the entire contents of which areincorporated herein by reference for all purposes.

FIELD OF THE INVENTION

The present invention is related to coding and decoding technology,especially related to a media decoding method based on cloud computingand decoder thereof.

BACKGROUND OF THE INVENTION

In the prior art, lossy compression is usually applied to a codingprocess, and within the same compressing standard, the bigger thecompression ratio is, the larger the loss is. Thus decoding qualitywhich is finally presented in a code stream compressed totally dependson compression methods and parameters used in the coding process.

In order to improve decoding quality of images or videos, mathematicalalgorithms are used to compensate lost pixels in lossy compression inthe prior art. However, if an excellent decoding effect is desired,those mathematical algorithms are usually very complicated, and thecomputing requirement of the decoding side is very high, so that it isnot practically applied. Moreover, since the essence of this method isestimating loss by using limited information of original images orvideos, in this situation, the decoding effect will not be very good forthose images or videos with poor quality.

Therefore, a new decoding method and decoder with lower computingcomplexity is required to improve decoding quality of various images orvideos.

SUMMARY OF THE INVENTION

A media decoding method based on cloud computing and decoder thereof areprovided by embodiments of the present invention, which are easy to useand applicable to a media of any form.

In an embodiment, a media decoding method based on cloud computingprovided by an embodiment of the present invention includes:

extracting representing features from a media code stream to be decoded;

searching in the cloud for a media object which has similar representingfeatures with the media code stream to be decoded by using a featurematching method and the representing features extracted;

filling, replacing and improving parts or segments of the media codestream to be decoded with whole or parts of the media object found.

In an embodiment, a decoder based on cloud computing provided by anembodiment of the present invention includes:

a feature extracting module, adapted to extract representing featuresfrom a media code stream to be decoded;

a feature matching module, adapted to search in the cloud for a mediaobject which has similar representing features with the media codestream to be decoded by using a feature matching method and therepresenting features extracted;

an image processing module, adapted to fill, replace or improve parts orsegments of the media code stream to be decoded with whole or parts ofthe media object found.

By using the technical scheme of the present invention, representingfeatures are extracted from a media code stream to be decoded, no matterwhat is the form of the media code stream, or what is the resolutionratio; then a media object having similar representing features with themedia code stream to be decoded is found in the cloud, and finallydecoding quality of the media code stream is improved by using thedetailed information in the media object found.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a flow chart of a media decoding method based oncloud computing provided by an embodiment of the present invention.

FIG. 2 illustrates a flow chart of a feature matching method provided byan embodiment of the present invention.

FIG. 3 illustrates a flow chart of a decoder based on cloud computingprovided by an embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

The embodiments of the present invention are described more fully hereinwith reference to the accompanying drawings, which form a part hereof,and which show, by way of illustration, specific exemplary embodimentsby which the invention may be practiced. This invention may, however, beembodied in many different forms and should not be construed as limitedto the embodiments set forth herein; rather, these embodiments areprovided so that this disclosure will be thorough and complete, and willfully convey the scope of the invention to those skilled in the art.Among other things, the present invention may be embodied as systems,methods or devices. The following detailed description should not to betaken in a limiting sense.

Throughout the specification and claims, the following terms take themeanings explicitly associated herein, unless the context clearlydictates otherwise. The phrase “in one embodiment” as used herein doesnot necessarily refer to the same embodiment, though it may.Furthermore, the phrase “in another embodiment” as used herein does notnecessarily refer to a different embodiment, although it may. Thus, asdescribed below, various embodiments of the invention may be readilycombined, without departing from the scope or spirit of the invention.

In addition, as used herein, the term “or” is an inclusive “or”operator, and is equivalent to the term “and/or,” unless the contextclearly dictates otherwise. The term “based on” is not exclusive andallows for being based on additional factors not described, unless thecontext clearly dictates otherwise. In addition, throughout thespecification, the meaning of “a,” “an,” and “the” include pluralreferences. The meaning of “in” includes “in” and “on”. The term“coupled” implies that the elements may be directly connected togetheror may be coupled through one or more intervening elements. Furtherreference may be made to an embodiment where a component is implementedand multiple like or identical components are implemented.

While the embodiments make reference to certain events this is notintended to be a limitation of the embodiments of the present inventionand such is equally applicable to any event where goods or services areoffered to a consumer.

Further, the order of the steps in the present embodiment is exemplaryand is not intended to be a limitation on the embodiments of the presentinvention. It is contemplated that the present invention includes theprocess being practiced in other orders and/or with intermediary stepsand/or processes.

FIG. 1 illustrates a flow chart of a media decoding method based oncloud computing provided by an embodiment of the present invention. Asshown in FIG. 1, the method includes following steps.

Step 100: representing features are extracted from a media code streamto be decoded.

The media code stream to be decoded includes images, videos, audios andother media forms which may be felt by users through any audible orvisual perception manner. Herein, the media code stream to be decodedmay be a kind of compressed code stream which is coded based on anycoding standard, or may be any primary media content captured by varioussensor devices, or even may be an incomplete media code stream.

In this step, representing features may be pre-extracted from the mediacode stream, and then be stored in the media code stream during a codingprocess; so when the media code stream is required to be decoded, therepresenting features are directly obtained from the media code streamat a decoding side. Those skilled in the art can understand that, therepresenting features also may be directly extracted from the media codestream at a decoding side without being pre-extracted and stored.

The representing features may include local features and/or globalfeatures. Specifically, the global features may describe colorhistograms, color moments or gray co-occurrence matrixes. These globalfeatures only represent global information of the image, and cannotrepresent objects contained in the image. However, local featuresextract interested contents from image contents, and have sufficientdescription and distinction ability to describe media features. Thelocal features are usually expressions describing one circular area.

Step 200: a media object which has similar representing features withthe media code stream to be decoded is found in the cloud by using afeature matching method and the representing features extracted.

There are a lot of media objects in the cloud, and each media object hascorresponding representing features. In order to improve efficiency ofsubsequent feature matching steps, these media objects may be chosen andtheir representing features may be pre-extracted in advance.

Step 300: parts or segments of the media code stream to be decoded arefilled, replaced or improved with whole or parts of the media objectfound to enhance the decoding quality.

First, the media object found is transformed by being zoomed or rotated;and then brightness of the media object transformed may be furtheradjusted, herein, the adjustments of brightness include contraststretching, mean value shift and so on; finally, the media objecttransformed and adjusted is used to fill, replace or improve parts orsegments of the media code stream to be decoded.

In an embodiment of the present invention, multiple media objects may beobtained by using a feature matching method. After being transformed andadjusted, loss of the media objects are estimated, and the optimal mediaobject is chosen or obtained according to a merging criterion, and isused to fill, replace or improve parts or segments of the media codestream to be decoded.

Even though a prior decoding process has been considered as sufficientlycomplete (for example, a TV completely decodes and restores ahigh-definition television media), the decoding effect could be furtherimproved by using the technical scheme of the present invention (forexample, a prior high-definition television media could be improved to aultra high-definition television media).

FIG. 2 illustrates a flow chart of a feature matching method provided byan embodiment of the present invention. As shown in FIG. 2, the featurematching method is provided to search in the cloud for a media objectwith reference to representing features extracted from a media codestream to be decoded, the method includes following steps.

Step 201: local features are extracted from the media code stream to bedecoded. In an embodiment of the present invention, SIFT(Scale-invariant feature transform) may be used to extract localfeatures from the media code stream. This step may be a repetition ofStep 100, so when the local features has already been obtained by Step100, Step 201 may be omitted.

Step 202: the local features extracted are compared with local featuresof a candidate media object in the cloud to obtain local feature pairs.Each local feature pair includes two identical or similar local featuresrespectively extracted from the media code stream to be decoded andobtained from the candidate media object. The two local features whichsimilarity degree is within a threshold range would be considered assimilar.

Step 203: geometric distributions of the local features corresponding tothe local feature pair are calculated respectively in the media codestream and the candidate media object.

Step 204: it is determined that whether the geometric distributions ofthe local features corresponding to the local feature pair, respectivelyin the media code and the candidate media object, are consistent. If thetwo geometric distributions are consistent, the candidate media objectis considered as the media object which has similar representingfeatures with the media code stream to be decoded.

For example, 1000 local features are extracted from a media code streamto be decoded and 800 local features are obtained from a candidate mediaobject, and 200 local feature pairs are obtained through featurecomparisons. Then geometric distributions of the local featurescorresponding to each of the 200 pairs are calculated respectively inthe media code stream to be decoded and the candidate media object. Ifthe geometric distributions of the local features corresponding to eachof the 200 local feature pairs, respectively in the media code and thecandidate media object, are considered as consistent, the candidatemedia object is considered as the media object which has similarrepresenting features with the media code stream to be decoded. Themedia object may carry more detail information than the media codestream to be decoded, so that the decoding quality of the media codestream can be improved by using the media object found.

In an embodiment of the present invention, in order to improve featurematching efficiency, local features of a media code stream to be codedmay be combined to obtain a global feature. A media object which has asimilar global feature with the media code stream to be decoded is foundin the cloud by using the feature matching method; then local featuresof the media code stream are compared with local features of thecandidate media object. By using this method, the feature matchingefficiency can be improved.

The feature matching method provided by an embodiment of the presentinvention is applicable to various situations. For example, when a mediacode stream to be decoded is a standard definition video, decodingquality of the standard definition video can be improved by obtainingsimilar contents from a media object found in the cloud, and then a highdefinition video is obtained.

Or, for media code streams formed after lossy compression, by referringto a media object found in the cloud, lost contents can be compensatedto achieve a decoding quality close to original images or videos. Inorder to ensure fidelity of the decoding process (extra signal beyondthe original images or videos is not produced), in an embodiment of thepresent invention, media objects (images or videos) which are used toimprove the decoding process are limited within a finite set only sharedby coders and decoders.

A decoder based on cloud computing is provided by an embodiment of thepresent invention. As shown in FIG. 3, the decoder includes: a featureextracting module, adapted to extract representing features from a mediacode stream to be decoded; a feature matching module, adapted to searchin the cloud for a media object which has similar representing featureswith the media code stream to be decoded by using a feature matchingmethod and the representing features extracted; an image processingmodule, adapted to fill, replace or improve parts or segments of themedia code stream to be decoded with whole or parts of the media objectfound.

In order to accelerate extracting speed of representing features, thefeature extracting module may include two units: an acquiring unit,adapted to directly acquire representing features, which have beenstored in the media code stream during a coding process, from the mediacode stream to be decoded; a calculating unit, adapted to calculaterepresenting features with reference to the media code stream to bedecoded.

Specifically, the feature matching module may further include: asearching unit for local feature pairs, adapted to compare localfeatures extracted from the media code stream with local features of acandidate media object in the cloud in order to obtain a local featurepair; a geometric distribution calculating unit, adapted to calculategeometric distributions of the local features corresponding to the localfeature pair respectively in the media code stream and in the candidatemedia object; a determining unit, adapted to determine whether thegeometric distribution of the local features corresponding to the localfeature pair in the media code stream is consistent with that in thecandidate media object; when they are consistent, consider the candidatemedia object as the media object which has similar representing featureswith the media code stream to be decoded. Besides, the feature matchingmodule may further include: a global feature matching unit, adapted tocombine local features of a media code stream to be decoded to obtain aglobal feature; and search in the cloud for a media object having asimilar global feature with the media code stream by using the featurematching method; then the searching unit for local feature pairs isadapted to compare the local features extracted by the featureextracting module with the local features of the candidate media objectsobtained by the global feature matching unit.

Specifically, the image processing module may include: a transformingunit for media objects, adapted to transform the media object found toconform the shape of the media code stream; an adjusting unit forbrightness, adapted to adjust brightness of the media objecttransformed; a merging unit, adapted to obtain the optimal media objectwith reference to multiple media objects transformed and adjusted, andfill, replace or improve parts or segments of the media code stream tobe coded with the optimal media object.

The above embodiments are only preferred embodiments of the presentinvention and cannot be used to limit the protection scope of thepresent invention. Those skilled in the art can understand that, thetechnical scheme of the embodiment may still be modified or partlyequivalently substituted; and the modification or substitution should beconsidered within the spirit and protection scope of the presentinvention.

The invention claimed is:
 1. A media decoding method based on cloudcomputing, comprising: extracting representing features from a mediacode stream to be decoded; searching in the cloud for a media objectwhich has similar representing features with the media code stream to bedecoded by using a feature matching method and the representing featuresextracted; filling, replacing and improving parts or segments of themedia code stream to be decoded with whole or parts of the media objectfound; wherein the representing features are local features; searchingin the cloud for a media object which has similar representing featureswith the media code stream to be decoded by using a feature matchingmethod and the representing features extracted comprises: comparing thelocal features with local features of a candidate media object in thecloud to obtain a local feature pair; calculating geometricdistributions of the local features corresponding to the local featurepair respectively in the media code stream and the candidate mediaobject; determining whether the geometric distributions of the localfeatures corresponding to the local feature pair in the media codestream, respectively in the media code and the candidate media object,are consistent; considering the candidate media object as the mediaobject which has similar representing features with the media codestream to be decoded, if the two geometric distributions are consistent.2. The method of claim 1, wherein, extracting representing features froma media code stream to be decoded comprises: directly acquiringrepresenting features, which have been stored in the media code streamduring a coding process, from the media code stream; or calculatingrepresenting features based on the media code stream to be decoded. 3.The method of claim 1, wherein, before comparing the local features withlocal features of a candidate media object in cloud to obtain a localfeature pair, the method further comprises: combining local features ofthe media code stream to obtain a global feature; searching in the cloudfor a candidate media object matching the global feature of the mediacode stream by using the feature matching method.
 4. The method of claim1, wherein, SIFT (Scale-invariant feature transform) is used forextracting the local features from the media code stream.
 5. The methodof claim 1, wherein, filling, replacing and improving parts or segmentsof the media code stream to be decoded with whole or parts of the mediaobject found comprises: transforming the media object found to conformthe shape of the media code stream; adjusting brightness of the mediaobject transformed; obtaining the optimal media object according to amerging criterion with reference to multiple media objects adjusted;filling, replacing or improving parts or segments of the media codestream to be coded with the optimal media object.
 6. The method of claim1, wherein, the media code stream comprises images, videos and audios.7. A tangible decoder based on cloud computing, comprising: a featureextracting module, adapted to extract representing features from a mediacode stream to be decoded; a feature matching module, adapted to searchin the cloud for a media object which has similar representing featureswith the media code stream to be decoded by using a feature matchingmethod and the representing features extracted; an image processingmodule, adapted to fill, replace or improve parts or segments of themedia code stream to be decoded with whole or parts of the media objectfound; wherein the feature matching module comprises: a searching unitfor local feature pairs, adapted to compare local features extractedfrom the media code stream with local features of a candidate mediaobject in the cloud to obtain a local feature pair; a geometricdistribution calculating unit, adapted to calculate geometricdistributions of the local features corresponding to the local featurepair respectively in the media code stream and in the candidate mediaobject; a determining unit, adapted to determine whether the geometricdistributions of the local features corresponding to the local featurepair in the media code stream, respectively in the media code and thecandidate media object, are consistent and consider the candidate mediaobject as the media object which has similar representing features withthe media code stream to be decoded, if the two geometric distributionsare consistent.
 8. The tangible decoder of claim 7, wherein, the featureextracting module comprises: an acquiring unit, adapted to directlyacquire representing features, which have been stored in the media codestream during a coding process, from the media code stream; and/or acalculating unit, adapted to calculate representing features withreference to the media code stream to be decoded.
 9. The tangibledecoder of claim 7, wherein, the feature matching module furthercomprises: a global feature matching unit, adapted to combine localfeatures of the media code stream to obtain a global feature; and searchin the cloud for a media object having a similar global feature with themedia code stream by using the feature matching method; the searchingunit for local feature pairs is further adapted to compare therepresenting features extracted by the feature extracting module withthe local features of the candidate media object obtained by the globalfeature matching unit.
 10. The tangible decoder of claim 7, wherein, theimage processing module comprises: a transforming unit for mediaobjects, adapted to transform the media object found to conform theshape of the media code stream; an adjusting unit for brightness,adapted to adjust brightness of the media object transformed; a mergingunit, adapted to obtain the optimal media object with reference tomultiple media objects transformed and adjusted, and fill, replace orimprove parts or segments of the media code stream to be coded with theoptimal media object.